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(54) High precision sub-pixel spatial alignment of digital images 



(57) A high precision sub-pixel spatial alignment of 
digital images, one from a reference video signal and 
another from a corresponding test video signal, uses an 
iterative process and incorporates spatial resampling 
along with basic correlation and estimation of fractional 
pixel shift. The corresponding images from the refer- 
ence and test video signals are captured and a test block 
is overlaid on them at the same locations to include tex- 
ture from the images. FFTsare performed within the test 
block in each image, and the FFTs are cross-correlated 
to develop a peak value representing a shift position be- 
tween the images. A curve is fitted to the peak and 



neighboring values to find the nearest integer pixel shift 
position. The test block is shifted in the test image by 
the integer pixel shift position, and the FFT in the test 
image is repeated and correlated with the FFT from the 
reference image. The curve fitting is repeated to obtain 
a fractional pixel shift position value that is combined 
with the integer pixel shift value to update the test block 
position again in the test image. The steps are repeated 
until an end condition is achieved, at which point the val- 
ue of the pixel shift position for the test block in the test 
image relative to the reference image is used to align 
the two images with high precision sub-pixel accuracy. 
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Description 

BACKGROUND OF THE INVENTION 

[0001] The present invention relates to the matching 
of digital images, and more particularly to high precision 
sub-pixel spatial alignment of digital images. 
[0002] Accurate spatial alignment of digital images is 
of fundamental importance to many applications. One 
such application is the determination of picture quality 
using an instrument, such as the PQA200 Picture Qual- 
ity Analyzer manufactured by Tektronix, Inc. of Wilson- 
ville, Oregon, USA, where images from a reference vid- 
eo signal are compared with corresponding images from 
a test video signal that is a processed version of the ref- 
erence video signal. The better the spatial alignment be- 
tween a reference image and a corresponding test im- 
age, the more accurate the determination of the amount 
of degradation of the test video signal due to the 
processing. 

[0003] At pages 480-484 of Computer Image 
Processing and Recognition (1979) published by Aca- 
demic Press, Ernest L. Hall describes a basic correlator 
method for template matching, which may be used for 
image alignment. Likewise J. J. Pearson et al in an article 
entitled Video-Rate Image Correlation Processor (IOCC 
1977) published in SPIE Vol. 119 Application of Digital 
Image Processing, pages 1 97-205, describe a variant 
of the basic correlator method that uses phase informa- 
tion. For digital images with integer pixel shifts these cor- 
relation methods may be very accurate. However for a 
broader class of shifts, which involves both integer and 
fractional pixel shifts, the precision of the sub-pixel 
alignment becomes a very serious matter. Pearson et 
al describe a method for estimating the sub-pixel shifts 
by quadratic interpolation using the correlation values 
in the neighborhood of a peak. However, since the quad- 
ratic interpolation function does not necessarily match 
the true image shifting function, the accuracy obtained 
by the quadratic interpolation is limited. 
[0004] What is desired is a high precision sub-pixel 
spatial alignment of digital images. 

BRIEF SUMMARY OF THE INVENTION 

[0005] Accordingly the present invention provides for 
high precision sub-pixel spatial alignment of digital im- 
ages using an iteration method and spatial resampling. 
A high precision sub-pixel spatial alignment of digital im- 
ages, one form a reference video signal and another 
from a corresponding test video signal, uses an iterative 
process and incorporates spatial resampling along with 
basic correlation and estimation of fractional pixel shift. 
The corresponding images from the reference and test 
video signals are captured and a test block is overlaid 
on them at the same locations to include texture from 
the images. FFTs are performed within the test block in 
each image, and the FFTs are cross-correlated to de- 
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velop a peak value representing a shift position between 
the images. A curve is fitted to the peak and neighboring 
values to find the nearest integer pixel shift position. The 
test block is shifted in the test image by the integer pixel 

5 shift position, and the FFT in the test image is repeated 
and correlated with the FFT from the reference image. 
The curve fitting is repeated to obtain a fractional pixel 
shift position value that is combined with the integer pix- 
el shift value to update the test block position again in 

to the test image. The steps are repeated until an end con- 
dition is achieved, at which point the value of the pixel 
shift position for the test block in the test image relative 
to the reference image is used to align the two images 
with high precision sub-pixel accuracy. 

75 [0006] The objects, advantages and other novel fea- 
tures of the present invention are apparent from the fol- 
lowing detailed description when read in conjunction 
with the appended claims and attached drawing. 

20 BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF 
THE DRAWING 

[0007] Fig. 1 is a block diagram view of a system for 
high precision sub-pixel spatial alignment of digital im- 

2S ages according to the present invention. 

[0008] Fig. 2 is an illustrative view of a comparison of 
an image from a reference video signal and an image 
from a corresponding lest video signal lo be spatially 
aligned with high precision sub-pixel accuracy accord- 

30 ing to the present invention. 

[0009] Fig. 3 is a flow diagram view of a high precision 
sub-pixel spatial alignment algorithm according to the 
present invention. 

[0010] Fig. 4 is an illustrative view of a cross-correla- 
35 tion surface showing a peak location for the high preci- 
sion sub-pixel spatial alignment algorithm according to 
the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

40 

[0011] Referring now to Fig. 1 an original, reference 
video signal from a source 1 0 is input to a video process- 
ing network 12. The video processing network 12 may 
include one or more compression/decompression pass- 

45 es. The output from the video processing network 12 is 
input to a decoder or similar device 14 to provide a 
source of a test video signal. The reference and test vid- 
eo signals are input to a video capture module 16 that 
captures corresponding images or frames from the two 

so video signals. The corresponding image pairs from the 
video capture module 16 are input to a high precision 
sub-pixel spatial alignment detect module 18 to deter- 
mine a position shift between the images of the refer- 
ence video signal and the images of the test video sig- 

55 nal. The position shift from the spatial alignment detect 
module 18 is input to a position shift module 20, which 
may be either in the reference or test signal path, to pre- 
cisely align the image pairs with sub-pixel accuracy from 
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the video captu re module 1 6 for further processing, such 
as by a picture quality analyzer 22 
[0012] In Fig. 2 the reference and test images 24, 26 
are shown having some texture 25, i.e., not a matte im- 
age. As shown in the test image after processing the 
texture 25 has been shifted horizontally and/or vertically 
An arbitrary test region or block 28 is formed, shown in 
this example as a rectangle, having a reference point 
Xo, Yo and is overlaid on both the reference and test 
images. For a rectangle as shown the reference point 
and horizontal and vertical extents AX, AY define the test 
region or block 28. The test region 28 is located where 
there is substantial texture 25 in the images. 
[0013] A high precision sub-pixel spatial alignment al- 
gorithm is shown in Fig. 3. An initialization module 30 
provides the corresponding reference and test images 
together with a test block to a correlation measurement 
module 32. The first step 34 in the initialization module 
30 controls the video capture module 16 to capture cor- 
responding reference and test images or frames from 
the reference and test video signals. The second step 
36 establishes a test block for overlaying on the respec- 
tive images in an area that has significant texture 25. 
The first step 38 of the correlation measurement module 
32 applies a fast Fourier transform (FFT) to the pixels 
of the images that lie within the test region. A cross-cor- 
relation step 40 is then performed in the FFT domain. 

FFT(corr) = FFT(ref)*FFT(tst) 

FFT" 1 (corr) produces correlation coefficients for every 
shift point within the test region, which may be repre- 
sented in the form of a surface 42 as shown in Fig. 4. 
The position of a peak 44 in the surface 42 indicates the 
amount of shift in position between the reference and 
test images. A curve-fit step 46 provides a nearest inte- 
ger pixel shift position for the peak 44 based upon the 
coefficients for the peak position and the positions up, 
down, left and right from the peak position. The pixel 
shift position from the correlation measure module 32 is 
input to an update shift position step 48. For the first 
iteration only the nearest integer pixel shift position is 
used. 

[001 4] A testing step 50 checks the amount of position 
shift determined from the update shift position step 48 
and the number of iterations against specified parame- 
ters to determine whether the spatial alignment algo- 
rithm is done. If the number of iterations is equal to the 
specified number, or the amount of update is less than 
a specified noise value, then the shift position value from 
the update step 48 is provided as an output pixel shift 
position and the algorithm ends. Otherwise a shift test 
block step 52 shifts the test region in the test image by 
the value from the update position step 48 so that the 
reference point is shifted to be X+X A , Y+Y A , where X A 
and Y A are the integer pixel shift position values. With 
the new position of the test region or block, the correla- 
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tion module 32 is applied again to obtain a fractional pix- 
el shift value in X and Y. In the FFT step 38 only the test 
image is processed after the initial iteration. The FFT for 
the reference image does not change. The fractional 

5 pixel shift value is used to update the shift position in 
the update step 48 so that the shift position becomes 
Xo+X A +X f , Yo+Y A +Y f . It has been found that for most 
practical applications two fractional pixel shift position 
iterations produce satisfactory results. The final shift po- 

10 sition value output from the spatial alignment detect 
module 18 is then used to spatially align the test and 
reference images as indicated above. 
[0015] Thus the present invention provides high pre- 
cision sub-pixel spatial alignment of digital images by 

15 using an iterative process to cross-correlate FFTs for a 
test block from each image in order to find a peak rep- 
resentative of a shift position between the images, then 
by fitting a curve to the correlation coefficients to deter- 
mine a nearest integer pixel shift position, updating the 

20 position of the test block in the test image and repeating 
until a specified number of iterations occurs or a change 
in fractional pixel shift position is less than a noise value. 



25 Claims 

1. A method of high precision sub-pixel spatial align- 
ment of digital images comprising the steps of: 

30 capturing a reference image from a reference 

video signal as one of the digital images and a 
corresponding lesl image from a lest video sig- 
nal as another of the digital images, where the 
test video signal represents the reference video 

35 signal after processing; 

overlaying an arbitrary test block on the refer- 
ence and test images at corresponding loca- 
tions that include texture; 
performing a fast Fourier transform for the por- 

40 tions of each image that lie within the test block 

to produce a reference FFT and a test FFT: 
cross-correlating the reference and test FFTs 
to produce a correlation surface having a peak 
correlation value, the position of the peak cor- 

45 relation value on the correlation surface being 

representative of a shift position between the 
reference and test images; 
fitting a curve to the peak and neighboring cor- 
relation values to determine an integer pixel 

so shift position; 

updating the position of the test block overlay- 
ing the test image by the integer pixel shift po- 
sition; and 

repeating the performing step for the test im- 
55 age, the correlating step for finding fractional 

pixel shift positions, and the updating step by 
adding the fractional pixel shift positions to the 
integer pixel shift position until an end condition 
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is achieved, the final position of the test block 
in the test image from the updating step relative 
to the initial position of the test block in the test 
image providing the high precision sub-pixel 
shift alignment between the reference and test s 
images. 

A method of high precision sub-pixel spatial align- 
ment of digital images comprising the steps of: 

10 

cross-correlating inthefrequency domain atest 
block for each of the digital images to produce 
a correlation surface having a peak correlation 
value at a particular position within the test 
block; is 
updating the test block in one of the digital im- 
ages as a function of the particular position; and 
repeating the cross-correlating and updating 
steps until a specified end condition is 
achieved, the final position of the test block in 20 
the one of the digital images relative to the other 
digital image representing the high precision 
sub-pixel spatial alignment between the digital 
images. 



EP 0 999 521 A2 



10 



REF 
VIDEO 



VIDEO 
PROCESSING 



TEST 
VIDEO 



VIDEO 
CAPTURE 



SPATIAL 
ALIGNMENT 
DETECT 



POSITION 



PQA 



FIG.1 



SHIFT 



22 

ORIGINAL ) 




24 

TEST ) 



(REFERENCE) 



FIG.2 




FIG.4 



EP 0 999 521 A2 



34- 



SPATIAL 
CORRELATION 
START 

I 



INITIALIZE 
ACQUIRE REF & 
TEST FRAMES 



>30 



36- 



ESTABLISH TEST 
BLOCK W/TEXTURE 



SHIFT TEST PEAK 
PER SHIFT 
LOCATION (INTEGER 
ONLY FIRST ITERATION) 




38 




40 




46 


50 


NO 



CORRELATION 

MEASURE 
PERFORM FFTS 
W/I TEST BLOCKS 
FOR REF k 
TEST FRAMES 



CROSS-CORRELATE 
FFTS TO FIND 
PEAK LOCATION 



QUAD CURVE-FIT 
FOR FRACTIONAL 
PIXEL POSITION 
ABOUT PEAK 
LOCATION 




UPDATE SHIFT 
POSITION (INTEGER 
SHIFT ONLY FIRST 
ITERATION) BASED 
UPON CURVE FIT 



FIG.3 



>32 



6 



